Namespaces

Within this specification, the following namespace prefix bindings are used:

Number Formatting

Within this document, decimal numbers will use a dot "." as the decimal mark.

Functional Requirements

The ADF Data Package API has the following functional requirements:

• package management: manage the contents of the data package
  • create empty files and remove files
  • read and write files (= stream to File Content; read , write , seek (random access) , append )
  • rename files (in File data description, optional: in HDF)
  • copy files (new File, clone File Content)
  • move files (File: adjust File Data description (parent), in HDF5 move Dataset, optional: move of File Content)
  • create folder
  • remove empty folders (convenience: folder with contents , recursive )
  • rename folder
  • move folder
  • copy folders (= copy folder plus its contents): do we really need that? (copy vs xcopy /s)
  • support links between files (symbolic links, hard links, aliases)
    • motivation: save storage space, support different organization concepts
  • import / export files and folders
• integrity management
  • create and update checksums (e.g. MD5, SHA1, etc.)
  • support signatures
  • support encryption
    • If yes, on which level? application level, ADF level, DP folder level, DP file level, (DC data cube level, TS level)
• support different file formats (e.g. WIFF, RAW, ..., JPEG, PNG, ..., PDF, ...)
• support different encodings
  • binary
  • text (character encoding, e.g. UTF-8; line feed, e.g. CR-LF)
  • image (native HDF5 format)
• data description management: manage the data description of the contents of the data package (File, File Content, Folder, Link(?), etc.)
  • inherent file meta data: file name, file size, timestamps (created, last modified, ...), file encoding, file creator, last file modificator, file extension
    • created: timestamp when created in ADF/HDF (original creation date: see provenance)
    • write protection
  • provenance metadata: who? from where? what? when? etc.
    • e.g. reproduce original filesystem structure
    • file attributes (e.g. read-only/executable/directory/archive/hidden)
  • file format: MIME type
  • file format specific metadata, e.g. image size, subformats, versions, codecs, etc.
• provide efficient audit trail of meta data (who? when? depending on GxP: why?)

Non-Functional Requirements

The ADF Data Package API has the following non-functional requirements:

• Data Consistency
• Space efficiency
  • compression: support compression of files
• Concurrency:
  • support concurrent read of one file within an ADF file
  • support concurrent write of different files within an ADF file
• Performance
• Embeddability into HDF5
• Synchronization between ADF-DP and ADF-DD
  • DP -> DD: If an update via DP is done, the metadata MUST be updated.
  • DD -> DP: If an update via DD is done, the data package MUST be updated.
  • DD -> DP: Prevent certain updates on DD that have side effects on DP, e.g. changing the relation between a file and a folder in DD

Compatibility

This feature has been designed to work with files from older version which do not support revisions. This means an ADF file could be written to by different versions of the ADF library (some supporting revisions, some not). And after writing with a version supporting revisions the full revision history of the written DpFile will be available. This will not break the feature.

Update Mechanism

The standard update mechanism is linked to the AuditRecordClosingEvent which triggers an update of the revision history of all DpFiles changed in the AuditRecord. Also, to support ADF files which might have been modified by an older Version of the ADF libary, on every access to a files revision history the last entry is checked against the AuditTrail of the DpFile. If the two entries match no further action will be taken.

If the AuditEntry is younger than the last revision entry all data needed for a new revision entry will be stored in a list and the next older AuditEntry will be checked. This continues until the AuditEntry has been found that matches the original last revision entry. Now the new revision entries can be created and numbered continuously.

If the ADF file can be modified these changes will be written directly to the AuditTrail model. Otherwise they are only written to an in-memory model and disgarded after use. This way even for older files which must not be changed the revision feature can be used - just slower since the AuditTrail has to be analysed every time.

Data Structure

All revision entries are connected to the file's Resource in the Audit model by the property pav:hasVersion. Also they form a doubly linked list with pav:nextVersion pointing to the next revision and pav:previousVersion pointing to the previsous revision. The revision node itself has a UUID based IRI.

The DpFile's Resource in the Audit model needs to have a rdf:type entry for prov:Entity and needs to be linked to the DataPackage node adf://dp by dct:hasPart, ore:aggregates and prov:hadMember. It also has an entry for pav:version with the current revision number as its value and points to the end of the doubly linked revision list with pav:previousVersion.

Values that should be present at each revision entry node:

• pav:version - Revision number, starting with 1 at the oldest revision
• adf-audit:hasAuditRecord - AuditRecord describing the change from this version to the next
• adf-dp:fileSize - File size in this revision
• adf-dp:modifiedBy - Modifying agent in this revision
• dct:modified - Modification time of this revision
• http://www.loc.gov/premis/rdf/v1#hasMessageDigest - The message digest of this revision

If values like adf-dp:modifiedBy or http://www.loc.gov/premis/rdf/v1#hasMessageDigest are not changed in an AuditRecord their values are ommited in that record. This also means they cannot be copied for the revision history. But when finally a change of the value happens the old value is noted in the AuditRecord and older revision entries (those missing the value) can be updated. If no change of the value has happened yet, its correct value can be found in the current data and be used to updated the revision history. The check and updated happens after every writing of an revision entry.

In case the DpFile has been deleted or recreated (delete and create in one AuditRecord) the HDF5 dataset representing the file in its old state will have been moved to an AuditTrail archive folder. So the revisions are no longer represented by the current state of the file. This is being addressed by adding a adf-dp:representedBy pointing to the adf-audit:archivedTo value of the AuditRecord's removal entry. This needs to be done for the previous revision entries until the first one that already has an adf-dp:representedBy entry.

Creating a Folder

The Data Package API MUST provide a method to create a folder in another folder of the Data Package. The method takes the name of the new folder as argument and returns a reference to the new folder.

Preconditions

The method MUST throw an exception, if a folder or file with this name already exists in the target folder.

The name of the folder MUST fulfil the interoperability requirements (e.g. avoid invalid characters, see subsection above). If not, the method MUST throw an exception.

Postconditions

The Data Package MUST contain the new Folder, which MUST have as parent the target Folder.

For the folder, an HDF5 group MUST have been created in the HDF5 group representing the target folder. HDF5 named objects like Groups or Datasets MUST use the identifier part of the UUID URN of the resource as specified in [[!rfc4122]]. The name of the new HDF5 group MUST be the UUID part of <folder-URI>, where folder-URI is a UUID URN using a type 4 (pseudo randomly generated) UUID.

Meta data of the folder MUST have been added to the ADF-QS via ADF-DD that fulfil the following requirements:

• Exactly one identifier ("id") is set: property dct:identifier, type xsd:string
• Exactly one folder name is specified with maximum length of 255 bytes is set not containing restricted characters: name "folder name", property dct:title, type xsd:string
• Exactly one creation time is set that MUST be the system time when the folder was created: name "created" property dct:created, type xsd:dateTime
• Exactly one system user is specified: name "created by", property dct:creator, type dct:Agent. If not exists, an instance of foaf:Person with dct:identifier <userid> is created.
• The creation date time is also set as the modification date time: name "modified", property dct:modified, type xsd:dateTime
• The system user responsible for the creation is also set as the modifier: name "modified by", property adf-dp:modifiedBy, type dct:Agent
• Exactly one folder is specified as the parent folder: name "parent folder", property dct:isPartOf, type adf-dp:Folder. The only exception here is the root folder. From the parent folder an inverse relation dct:hasPart and ldp:contains are set, referencing the newly created folder. The values of dct:modified and adf-dp:created of the parent folder are updated to hold the same values as the newly created folder.
• Exactly one URI is set for physical representation of the folder: name "physical representation", property adf-dp:representedBy, type hdf:Group. The URI is built from the absolute path of the folder within the HDF5 file.

Non-Functional Requirements

The method has to be transactional.

Accessing a Folder

The ADF-DP API MUST provide a method to access a given folder in a Data Package. The folder MAY be given by URI, by absolute path or by relative path. The method takes the name resp. the URI of the folder as argument and returns a reference to this folder, if it exists.

Preconditions

The method MUST throw an exception, if the folder given by URI or by path does not exist.

Postconditions

The folder returned by this method MUST fulfil the following requirements for accessing a folder:

• Exactly one identifier is specified: name "id", property dct:identifier, type xsd:string
• Exactly one folder name is specified with maximum length of 255 bytes is set not containing restricted characters: name "folder name", property dct:title, type xsd:string
• Exactly one creation time is specified that MUST be the system time when the folder was created: name "created" property dct:created, type xsd:dateTime
• Exactly one system user is specified: name "created by", property dct:creator, type dct:Agent.
• Exactly one date of the last modification is specified: name "modified", property dct:modified, type xsd:dateTime
• Exactly one responsible for the last modification is specified: name "modified by", property adf-dp:modifiedBy, type dct:Agent
• At most one parent folder is specified: name "parent folder", property dct:isPartOf, type adf-df:Folder.

Updating a Folder

When updating a folder, the meta data of this folder MUST fulfil the following requirements:

• Exactly one identifier is specified: name "id", property dct:identifier, type xsd:string
• Exactly one folder name is specified with maximum length of 255 bytes is set not containing restricted characters: name "folder name", property dct:title, type xsd:string
• Exactly one creation time is specified that MUST be the system time when the folder was created: name "created" property dct:created, type xsd:dateTime. May not be changed.
• Exactly one system user is specified: name "created by", property dct:creator, type dct:Agent. May not be changed.
• Exactly one date of the last modification is specified: name "modified", property dct:modified, type xsd:dateTime
• Exactly one responsible for the last modification is specified: name "modified by", property adf-dp:modifiedBy, type dct:Agent
• At most one parent folder is specified: name "parent folder", property dct:isPartOf and ldp:member, type adf-df:Folder. From the parent folder an inverse relation dct:hasPart and ldp:contains MUST be set, referencing the updated folder. The values of dct:modified and adf-dp:modifiedBy of the parent folder are updated to hold the same values as the newly created folder.
• Exactly one URI is set for physical representation of the folder: name "physical representation", property adf-dp:representedBy, type hdf:Group. The URI is built from the absolute path of the folder within the HDF5 file.

Listing the Content of a Folder

The ADF-DP API SHOULD provide a method to list the contents of a folder in a given Data Package. The method MUST return an empty list, if the folder does not have any contents.

The metadata of files and folders to load is based on the requirements for accessing files and accessing folders.

Deleting a Folder

The ADF Data Package API MUST provide a method to remove an empty folder within a Data Package. If the given folder does not exist in the Data Package, the method MUST throw an exception. If the folder is not empty, the method MUST throw an exception. If the folder is referenced by other predicates in ADF-QS, it MUST NOT be removed and the method MUST throw an exception.

Here "removal" means that the folder is removed from the Data Package - it does not need to be removed physically in HDF! Further meta data of the folder are removed from the ADF-QS.

Creating a File

When creating a file, a UUID URN URI MUST be generated (file-URI) and the class of the file stated (adf-dp:File and ldp:Resource) in the ADF-QS. In addition the following requirements for the class adf-dp:File MUST be fulfilled at creation time:

• Exactly one identifier is specified: name "id", property dct:identifier, type xsd:string
• Exactly one folder name is specified with maximum length of 255 bytes is set not containing restricted characters: name "folder name", property dct:title, type xsd:string
• Exactly one creation time is specified that MUST be the system time when the folder was created: name "created" property dct:created, type xsd:dateTime
• Exactly one system user is specified: name "created by", property dct:creator, type dct:Agent.
• Exactly one date of the last modification is specified: name "modified", property dct:modified, type xsd:dateTime
• Exactly one responsible for the last modification is specified: name "modified by", property adf-dp:modifiedBy, type dct:Agent
• Exactly one file format is specified: name "file format", property dct:format, value from [[MediaTypes]] (default value is http://purl.org/NET/mediatypes/application/octet-stream)
• Exactly one size of the file content in bytes is specified: name "file size", property adf-dp:fileSize, type xsd:long
• At most one charset such as UTF-8 (default value) is specified: name "charset", property adf-dp:charset, type xsd:string. The character set (encoding) is required, if file format is text.
• At most one line separator (one of "CR", "LF", "CRLF", "NEL" (see [[!UTR-13]])) is specified: name "line separator", property adf-dp:lineSeparator, type xsd:string. The line separator is required, if file format is text.
• At most one parent folder is specified: name "parent folder", property dct:isPartOf and ldp:member, type adf-df:Folder
• Exactly one path is specified: name "physical representation", property adf-dp:representedBy, type hdf:Dataset

Accessing a File

The ADF-DP API MUST provide a method to access a given file in a Data Package. The file MAY be given by URI, by absolute path or by relative path. The method takes the name resp. the URI of the file as argument and returns a reference to this file, if it exists.

Preconditions

The method MUST throw an exception, if the file given by URI or by path does not exist.

Postconditions

The file returned by this method MUST fulfil the following requirements:

• Exactly one identifier is specified: name "id", property dct:identifier, type xsd:string
• Exactly one folder name is specified with maximum length of 255 bytes is set not containing restricted characters: name "folder name", property dct:title, type xsd:string
• Exactly one creation time is specified that MUST be the system time when the folder was created: name "created" property dct:created, type xsd:dateTime
• Exactly one system user is specified: name "created by", property dct:creator, type dct:Agent.
• Exactly one date of the last modification is specified: name "modified", property dct:modified, type xsd:dateTime
• Exactly one responsible for the last modification is specified: name "modified by", property adf-dp:modifiedBy, type dct:Agent
• Exactly one file format is specified: name "file format", property dct:format, value from [[MediaTypes]] (default value is http://purl.org/NET/mediatypes/application/octet-stream)
• Exactly one size of the file content in bytes is specified: name "file size", property adf-dp:fileSize, type xsd:long
• At most one charset such as UTF-8 (default value) is specified: name "charset", property adf-dp:charset, type xsd:string. The character set (encoding) is required, if file format is text.
• At most one line separator (one of "CR", "LF", "CRLF", "NEL" (see [[!UTR-13]])) is specified: name "line separator", property adf-dp:lineSeparator, type xsd:string. The line separator is required, if file format is text.
• At most one parent folder is specified: name "parent folder", property dct:isPartOf and ldp:member, type adf-df:Folder

Updating a File

When updating a file, the following meta data of the class adf-dp:File MUST be fulfilled at modification time and updated in the ADF-QS.:

• Exactly one identifier is specified: name "id", property dct:identifier, type xsd:string. The identifier MAY NOT be changed (MUST be exactly the UUID part of the URI).
• Exactly one folder name is specified with maximum length of 255 bytes is set not containing restricted characters: name "folder name", property dct:title, type xsd:string
• Exactly one creation time is specified that MUST be the system time when the folder was created: name "created" property dct:created, type xsd:dateTime. May not be changed.
• Exactly one system user is specified: name "created by", property dct:creator, type dct:Agent. MAY NOT be changed.
• Exactly one date of the last modification is specified: name "modified", property dct:modified, type xsd:dateTime. MUST be set to the current system time.
• Exactly one responsible for the last modification is specified: name "modified by", property adf-dp:modifiedBy, type dct:Agent. The user MUST be set to the current system user, the agent SHOULD be an instance a foaf:Person with dct:identifier = <username>
• Exactly one file format is specified: name "file format", property dct:format, value from [[MediaTypes]] (default value is http://purl.org/NET/mediatypes/application/octet-stream)
• Exactly one size of the file content in bytes is specified: name "file size", property adf-dp:fileSize, type xsd:long. MAY need to be changed.
• At most one charset such as UTF-8 (default value) is specified: name "charset", propertyadf-dp:charset, type xsd:string. The character set (encoding) is required, if file format is text.
• At most one line separator (one of "CR", "LF", "CRLF", "NEL" (see [[!UTR-13]])) is specified: name "line separator", property adf-dp:lineSeparator, type xsd:string. The line separator is required, if file format is text.
• At most one parent folder is specified: name "parent folder", property dct:isPartOf and ldp:member, type adf-df:Folder. The requirements for updating a folder MUST be fulfilled at modification time for the parent folder.

The ADF-DP API MUST provide a method to write bytes into a given File within a Data Package, using a streaming API.

The method MUST support the following use cases:

• If the given file does not exist in the Data Package, the method MUST create it (cp. StandardOpenOption.CREATE in Java).
• If a file or folder with the given name already exists in the folder, the method MUST throw an exception (cp. StandardOpenOption.CREATE_NEW in Java).
• If the given file does already exist, the method MUST to support the following options:
  • Overwrite the existing file, starting at the beginning of the file (cp. StandardOpenOption.TRUNCATE_EXISTING in Java).
  • Append to the existing file, starting at the end of the file (cp. StandardOpenOption.APPEND in Java).

Non-Functional Requirements

The method MUST support variable length files. The chunk size MUST be configurable and preset to a reasonable value. When using a chunked HDF5 dataset and exceeding the size of the current chunk, a new chunk MUST be allocated and used implicitly. The method MAY support fixed length files in a later release of ADF, e.g. for performance optimization of read-only files (using HDF5 opaque datatype).

Postconditions

In case of CREATE, a variable length HDF5 dataset using the HDF5 VLEN type based on byte MUST have been created, representing the new File. The bytes MUST have been written into this HDF5 dataset. Meta data of the file MUST have been added to the ADF-QS via ADF-DD as defined in Creating a File.

In the non-exceptional case of CREATE_NEW, the bytes MUST have been written into the existing HDF5 dataset representing the file, starting at the first index position. Meta data of the file MUST have been updated in the ADF-QS via ADF-DD according to the requirements for updating a file.

In case of TRUNCATE_EXISTING, the bytes MUST have been written into the existing HDF5 dataset representing the file, starting at the first index position. Meta data of the file MUST have been updated in the ADF-QS via ADF-DD according to the requirements for updating a file.

In case of APPEND, the bytes MUST have been written into the existing HDF5 dataset representing the file, starting at the last index position. Meta data of the file MUST have been updated in the ADF-QS via ADF-DD according to the requirements for updating a file.

Design Decisions

Deleting a File

The ADF Data Package API MUST provide a method for the removal of files. Here "Removal" means that the file is marked as removed, but not actually physically removed - this holds for both, its HDF5 representation and its RDF representation.

In order to be able to remove a file, it MUST be ensured that it is not referenced by a resource in ADF-QS.

Import

The import service creates ADF-DpFiles and ADF-DpFolders according to the given source and MUST preserve the hierarchy of the source. Additionally, the following retrieval and source meta-data are stored:

• Retrieving agent, pav:retrievedBy, type dct:Agent
• Retrieval time, pav:retrievedBy, type xsd:dateTime
• Meta-data of the source, pav:retrievedFrom
  • File name, dct:title, type xsd:string
  • Creator, pav:createdBy, type dct:Agent
  • Creation time, pav:createdOn, type xsd:dateTime
  • Last modified, dct:modified, type xsd:dateTime
  • Path to the source, adf-dp:URL, type xsd:anyURI
  • Meta-data of the host, adf-dp:path
    • host name, adf-dp:hostname, type xsd:string
  • Path of the corresponding ADF-DpFile or ADF-DpFolder, adf-dp:path, type xsd:string
  • Path to the parent ADF-DPFolder, adf-dp:path, type dct:identifier

Preconditions

The method MUST throw an exception if either the source path or the target ADF-DPFolder do not exist.

Export

The export service restores the files and folders of the original source to the specified target. Symbolic links are replaced by their target.

Preconditions

The method MUST throw an exception if either the source ADF-DpFile or ADF-DPFolder or the target path do not exist.